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Background / Context: Student vocabulary growth needs to be a priority of instruction if 
schools are to meet the goal of helping every student become college and career ready. With 
more challenging expectations for academic achievement, the importance of vocabulary to 
reading cannot be ignored. Vocabulary is a significant correlate of concurrent and future reading 
achievement (Cunningham & Stanovich, 1997; Storch & Whitehurst, 2002). Yet, young children 
who struggle with reading do not develop adequate vocabularies, and therefore do not read well; 
because they are not reading well, they are not learning vocabulary (Aamoutse & van Leeuwe, 
1998; Chall, Jacobs, & Baldwin, 1990; Cunningham & Stanovich, 1997; Dickinson & Tabors, 
2001; Juel, 1988; Moats, 2001; White, Graves & Slater, 1990). During third grade, for example, 
above average readers make 31 percent gains in vocabulary; below average readers make gains 
of only 12 percent (Chall, Jacobs, & Baldwin, 1990). 

Educational practices are needed to promote strong vocabulary growth so that vocabulary 
can be both the cause and result of successful reading. Vocabulary interventions need to begin 
early and continue for a sustained period of time (Beck & McKeown, 2007a; Biemiller, 2003; 
Foorman, Seals, Anthony & Pollard-Durodola, 2003). According to theoretical and empirical 
research, there are three components of vocabulary instruction that are needed to boost 
vocabulary growth and improve reading. A number of literature reviews have concluded that 
successful vocabulary instruction includes: (1) both definitional and contextual information, (2) 
more than one or two exposures to each word, and (3) engagement of students in deep processing 
about word meaning and use (Baumann, Kameenui, & Ash, 2003; Beck, McKeown & Kucan, 
2002; Blachowicz & Fisher, 2000; Mezynski, 1983; Nagy & Scott, 2000; Rupley & Nichols, 
2005; Stahl & Fairbanks, 1986). 

Elements of Reading®: Vocabulary (EOR-V) (Beck & McKeown, 2004) is an 
instructional intervention that was designed to incorporate these three components and be used 
daily by classroom teachers to supplement core reading instruction. As EOR-V was developed in 
parallel fashion for use in consecutive grades (K-5), it provides an opportunity to examine the 
effects of early and sustained vocabulary instruction on children’s vocabulary and reading 
comprehension. Prior research supports the promise of EOR-V for positively impacting 
vocabulary and reading comprehension (Apthorp, 2006; Beck & McKeown, 2007b; Beck et ah, 
1982; McKeown, Beck, Omanson, & Perfetti, 1983; McKeown, Beck, Omanson, & Pople, 

1985), but a randomized control trial of the intervention with appropriate treatment of clustering 
effects has not yet been conducted. 

Purpose / Objective / Research Question / Focus of Study: The purpose of this study was to 
provide unbiased estimates of the effects of robust vocabulary instruction as instantiated in EOR- 
V in schools serving children from low-income households. Vocabulary researchers recognize 
that word learning does not occur easily or quickly (McKeown, Beck, Omanson & Pople, 1985; 
White, Graves & Slater, 1990). Children need early, more and continuous robust vocabulary 
instruction to develop a vocabulary of sophisticated words (Beck & McKeown, 2007a). The 
research to be presented addressed this need by testing the effects of a full, one- and two-year 
implementation of EOR-V on both vocabulary and reading comprehension. The specific purpose 
of the proposed presentation is to report estimates of the proximal effects of EOR-V on 
vocabulary and listening or reading comprehension as evaluated at the end of the first year of the 
two-year intervention. 

Setting: Forty-six elementary schools located in four districts/areas in the South-eastern United 
States agreed to participate in this study. Two schools dropped from the study after random 
assignment leaving 44 schools, all but one of which met Title 1 eligibility requirements. The 
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average percent of students receiving free/reduced lunch across the schools was 72% ranging 
from 43 to 96%. Approximately 41% of participant schools were in suburban locations, with 
31% from rural locales and towns, and 27% within small to large cities. 

Population / Participants / Subjects: Study participants included all general education 
classroom teachers and their students in kindergarten and grades one, three and four in Year 1 
(2008-2009) and all classroom teachers and their students in grades one, four and five in Year 2 
(2009-2010). Participants include 753 teachers and 9,313 students (kindergarten, 1*^', 3’^‘^ and 4* 
grades) in Year 1 of the study. 

Intervention / Program / Practice: The EOR-V program supplements any core kindergarten 
through fifth grade reading program using 15- to 20-minute daily lessons that progress in a five- 
day sequence from read-alouds to viewing photo cards to increasingly challenging discussion 
and activity prompts to a weekly quiz. The activities are designed to (1) provide children both 
definitional and contextual information about each vocabulary word, (2) provide more than one 
or two exposures to each word, and (3) engage students in deep processing about word meaning 
and use. Lessons focus on rather sophisticated, unfamiliar words that have application across 
multiple domains - labeled Tier 2 words. Tier 2 words provide more refined labels for familiar 
concepts. For example, “benevolent” can replace a common word such as “kind” (Beck, 
McKeown & Kucan, 2002). A new set of seven Tier 2 words are taught in each of the 24 EOR-V 
weekly lessons. Teachers implement the program using the five-day, weekly lesson plans and 
materials provided in the EOR-V teacher’s guide and classroom kit. 

Reading coaches from each school were trained in the use of EOR-V and subsequently 
trained and supported treatment teachers in the use of EOR-V. Training occurred in October and 
November 2008. EOR-V materials arrived in two districts/areas for 12 schools in October and 
November 2008 and in January 2009 in the other two districts/areas for 32 schools. Thus, there 
was a late start to implementation of EOR-V in the majority of schools. 

Research Design: The study used a cluster randomized trial design in which schools were the 
unit of assignment. Schools were blocked by district/area and randomly assigned to either 
Primary or Intermediate Intervention. In the Primary Intervention schools, the primary grades 
were the treatment group and the intermediate grades were the control group. In the Intermediate 
Intervention schools, the reverse was the case with the primary grades as the control group and 
the intermediate grades as the treatment group. As shown in Table 1 (Research Design), 
orthogonal to group assignment, are two studies: Study 1 (Primary Grades) and Study 2 

rd 

(Intermediate Grades). Kindergartners and 1st graders in Primary Intervention schools and 3 
and 4'*’ graders in Intermediate Intervention schools received EOR-V for two consecutive years. 

In the control classrooms for two consecutive years, vocabulary was taught business as usual. 
Data Collection and Analysis: To collect data on EOR-V implementation fidelity and 
programmatic differentiation between treatment and control conditions, teacher surveys, site 
visits, and audio-recording of lessons were used. To measure vocabulary and comprehension, 
pre-tests and Year 1 posttests were administered to students. 

To control for prior academic performance, students were administered the SAT- 10 as a 
pre-test prior to the start of the intervention. Students completed selected SAT- 10 subtests. 
Listening or Reading, that reflect achievement of the key outcomes in the study. The reliability 
estimates for the subtests are high enough to distinguish groups of students (frequently in the 
0.80s, but ranging from 0.53 to 0.93; Carney, 2005; Morse, 2005). 

To measure proximal effects, students were administered researcher-developed Tests of 
Instructed Word Knowledge in vocabulary and comprehension (TOIW-V and TOIW-C) in the 
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spring of Year 1. The TOIW-V is an orally group- administered instrument developed 
specifically to assess recognition and understanding of EOR-V Y\&x-2 words. Each grade level of 
the TOIW-V assessed students’ knowledge of 20 Tier-2 words randomly sampled from the first 
15 lessons of EOR-Vocabulary . The TOIW-V was based on a verbal task format that two 
members of the research team developed for an earlier study (Beck and McKeown, 2007) and 
that has been used by Coyne’s research group (Coyne, McCoach, Loftus, Zipoli, & Kapp, 2009). 
The test is designed to engage students in recognition of word meaning and judging correctness 
of word-use based on understanding of word meaning. Students are asked to respond yes or no to 
four questions about each word. Two questions (one true and one false) ask whether a presented 
meaning matched a given word (e.g., “Does extraordinary mean very special?” “Does 
extraordinary mean very hungry?”). Two questions ask whether a brief context is appropriate for 
a word (e.g., “Would it be extraordinary to see a monkey at the zoo?” “Would it be extraordinary 
to see a monkey teaching school?”). 

The comprehension test (TOIW-C) was developed to engage students in listening to or 
reading brief narratives with an emphasis on using their knowledge of target words in the 
passage to understand what was happening and why (Beck and McKeown, 2007). For example, 
in one passage, two children want to do something humane to help families who lost their 
belongings in a fire. So they make an ambitious plan and collect items from their neighbors, who 
graciously donate so much that the children’s garage resembles a department store. Looking at 
all they’ve collected, the children feel so good they get giddy. The test at each level comprised 
four passages, each using five target words from the first 15 lessons of EOR-V program. Each 
passage was a brief narrative involving one, two or three characters at school, at someone’s 
home, or in a community setting (e.g., the bakery). Five multiple-choice, “what,” “how” and 
“why” comprehension questions followed each passage. The questions asked about important 
events, descriptions, and feelings in the passage and addressed concepts expressed by the target 
words. The kindergarten and first grade TOIW-C was administered orally with each passage 
read aloud twice. The students answered questions by selecting one of three pictures which they 
viewed as the question was asked orally (e.g., “Which picture shows what the town did on the 
day of the parade?”). The third and fourth grade TOIW-C was administered as a paper-and- 
pencil assessment; students read passages and answered written multiple choice questions on a 
page following each passage. 

An item analysis was conducted on the TOIW-V and TOIW-C at each grade level. A 
minimum point-biserial correlation (pb(r) > 0.10) was used to eliminate items that showed weak 
associations with student’s overall score. Item analyses were conducted separately on each 
instrument. As an index of discrimination, items (question responses) with a point biserial 
correlation (pbr) < 0.10 were removed from the calculation of the composite score for that 
assessment for grades 1, 3, and 4. The results of these analyses are presented in Table 2. Due to 
low overall pb(r) scores on the grade-K instruments, all items were retained. Reliability 
coefficients (alpha) for all instruments ranged from 0.41to 0.87. The lowest reliability was found 
on the K and grade- 1 TOIW-Comprehension instruments. 

To examine proximal effects in Year 1, the average effects of EOR-V on instructed word 
knowledge and comprehension were estimated via 2-level hierarchical models with students 
(Level 1) nested within schools (Level 2). The model was run separately using TOIW-V and 
TOIW-C scores as the dependent variable at each grade level. 

Findings / Results (Fidelity of Implementation): Researchers examined fidelity of 
implementation of EOR-V with regard to two aspects of fidelity: dosage and adherence. Each of 
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these aspects of fidelity is defined below. The measurement of each aspect also is explained and 
results of the fidelity evaluation are presented. In addition, researchers examined generic fidelity 
by focusing on program differentiation. Researchers sought to verify that the unique features of 
the intervention were implemented and evaluated the extent to which EOR-V instruction differed 
from the counterfactual instruction. 

Dosage. Typically, “dosage is defined as the frequency and duration of an intervention” 
(Hamre, et ah, 2009, p. 2). Dosage was measured using number of EOR-V lessons completed. At 
each grade level, the EOR-V classroom kit includes 24 lessons intended for one full-school year 
of implementation. On average, classrooms implemented 12 lessons during Year 1 (see Table 
3). Low average dosage was associated with the late start to implementation. 

Adherence. Adherence is the extent to which program elements are used as intended or 
prescribed (Hamre et ah, 2009). The EOR-V teacher’s guide presents, and training for this study 
reinforced, 10 lesson elements to be implemented for each 5-day lesson. In Day I, two elements 
are intended for use to introduce words (Read- aloud; Explain meaning). Across Days 2, 3 and 4, 
five elements are intended for use to provide multiple exposures to and opportunities for use of 
the words (Word Snapshots, Word Chat, Student Book, Graphic Organizers, and Writing). On 
Day 5, the two elements involved Review and Assessment. The tenth element of the EOR-V 
program. Depth of Processing, is intended for use across all days; teachers are expected to ask 
students “why” follow-up questions to prompt thinking and reasoning. 

To evaluate adherence, researchers used data collected in the weekly EOR-V lesson-logs. 
The lesson logs asked teachers whether or the extent to which they implemented an element. For 
each element, the criterion for Adequate Fidelity was defined as a response of “yes, 
implemented,” or “implemented to a great extent.” Three lesson logs were randomly selected 
(one from each of the first quarters of the 24 lesson set). The percent of teachers meeting the 
adequate fidelity criterion per lesson was computed and averaged across the three lessons to 
report percent of teachers adhering to the element’s fidelity as prescribed. 

More than 80 percent of teachers demonstrated Adequate Adherence Fidelity for six of the 
10 program elements (see Table 4): Read-aloud, Student Book, Graphic Organizers, Review, 
Assessment, and the use of photo cards. Word Snapshots. 

Fidelity for two of the elements. Word Chat, and Depth of Processing (i.e., ask “why” 
follow-up questions), was adequate for roughly three-quarters of teachers (see Table 4). Fidelity 
for Writing was lower, except for grade 4. Teachers reported that they often did not have time for 
the writing activity. At grade 4, however, writing is an important part of the curriculum; in fact, 
there is a state- wide writing test for all 4^'’ grade students. The 4* grade teachers were observed 
and frequently reported that they used the EOR-V writing prompts as preparation for the 4* 
grade state assessment. For the final element. Explain Meaning, between 47 and 63 percent of 
teachers (L‘ and 4* grade, respectively) demonstrated adequate adherence (see Table 4). 

Program Differentiation. Based on earlier research and development in robust vocabulary 
instruction (Beck, Perfetti, & McKeown, 1982; McKeown, Beck, Omanson, & Perfetti, 1983; 
McKeown, Beck, Omanson, & Pople, 1985), the EOR-V authors included word chat prompts in 
the EOR-V teachers’ guides and lesson plans that are intended to engage students in deep 
processing of words. The word chat prompts ask students to make decisions about which 
contexts fit a word’s meaning, compare words, and relate words to one another. For example, the 
directions prompt students to discuss whether a swim coach could verify that a swimmer’s dive 
was audacious, or, why owls have a reputation for being subtle birds. 
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The word chat prompts are a unique feature of EOR-V. Using Year 1 data, researchers 
evaluated the extent to which instructional conversations with students in both treatment and 
control classrooms prompted deep processing of words during a vocabulary lesson. A random 
sample of one teacher per grade and school was asked to audio-record a vocabulary lesson in 
April 2008. The sample included both treatment and control teachers. Margaret McKeown and 
Isabel Beck developed a Depth of Processing Taxonomy to analyze the audio-taped lessons. 
Teacher questions were categorized into one of six categories (Read/repeat, recall, connect, 
generate/locate, integrate, and explain reasoning. The McKeown and Beck research team read 
and scored the audio-taped transcripts blind (without treatment or control identification) using 
the Depth of Processing Taxonomy. Distinct and statistically significant differences in the 
quality of the treatment (EOR-V) and control teachers’ question/answer interactions were 
documented, especially at the lowest and highest levels. Control classrooms were characterized 
by processing at literal levels, questions that required little more than recall of instructed 
definitions. Treatment classrooms had substantially less of such questions, and showed 
significantly more processing at the higher levels, with questions that asked students to explain 
examples of word use or their reasons for choosing contexts as appropriate for new words. 
Control classrooms generally lacked this type of processing. 

Findings / Results {Proximal Effects on Vocabulary and Comprehension): As shown 
in Table 5, the students in the treatment condition performed significantly higher on both 
TOIW-V and TOIW-C. As also can be seen in Table 5, effect sizes were greater for 
estimating the impact of EOR-V on vocabulary than on comprehension. Using the 
Benjamini-Hochberg Correction for multiple comparisons, all results remained 
statistically significant at the 0.05 level. 

Conclusions: Results for adherence to program elements demonstrate that not all elements were 
used to a high level. However, lessons were still significantly different from control classrooms 
as demonstrated by the Depth of Processing analysis of classroom discussion. Results for dosage 
indicated that many teachers used only half the available lessons. Yet, despite the low dosage 
and the less than ideal adherence fidelity, the intervention still seemed effective enough to 
promote significant differences in achievement in both knowledge of the target words and 
comprehension using the target words at both primary and intermediate grade levels. The 
magnitude of the present proximal effects on vocabulary was as great as or greater than the 
average proximal effect reported for vocabulary interventions (0.79) in a recent meta-analysis 
(Elleman, Eindo, Morphy & Compton, 2009). 

Eindings confirm recommendations for vocabulary instruction that provides multiple 
contexts and engages students’ processing of meaning to achieve the goals of increased 
vocabulary knowledge and comprehension. The findings demonstrate that such 
recommendations apply to primary and intermediate grades students. Eurther, findings suggest 
that expected knowledge and comprehension results can occur even when implementation is less 
intense than usually prescribed. The limitations are related to measurement and implementation 
issues. The low reliability estimates and pb(r) scores of the TOIW-C measures in kindergarten 
and U' grade indicate that at these grade levels, the intended outcomes may not be well- 
represented by the scores. The levels of dosage and adherence fidelity obtained may have limited 
discovering the extent to which robust vocabulary instruction can affect comprehension. 
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Appendix B. Tables and Figures 
Table 1. Research Design 
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Table 2. Reliability of Test of Instructed Words (TOIW) 
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Table 3. 



Implementation Fidelity Results by Grade: Dosage 
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Table 4. 



Implementation Fidelity Results by Grade: Adherence 







Percentage of Treatment Teachers Demonstrating Adequate 
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Table 5 



Year 1 Proximal Effects of EOR-V 
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